Stable Diffusion 1.5 was initially trained on a data set, called the LAION 5B data set, that contained nearly 6 billion images. Despite that, there are many concepts that Stable Diffusion struggles with. You can add concepts or styles to Stable Diffusion by training a low-rank matrix (abbreviated as LoRA) and adding it to the original Stable Diffusion matrix. There are some concepts that I have previously tried to get Stable Diffusion to create that I wasn't entirely satisfied with. Sometimes the images were still compelling, just not what I had originally asked for. Other times the images were just junk. So I decided to try training a LoRA.
For my first attempt, I decided to create a LoRA that mimics the style of David Goodsell. Goodsell is a structural biologist at both Scripps and Rutgers. Outside of his scientific research career, Goodsell is known for producing highly intricate water color paintings of cells, viruses, etc. at the molecular level.
Though I didn't say so at the time, my post Psychedelic Micrographs I was an attempt to create images in Goodsell's style. To train the LoRA, I used a set of Goodsell's paintings (available at https://pdb101.rcsb.org/sci-art/goodsell-gallery) that he has released under the CC-BY-4.0 license, which allows for the images (and style) to be adapted.
Here follows some of the images that I was able to generate using the LoRA that I trained. The captions indicate the subject of the prompt with the Stable Diffusion model that was used in parentheses.
Virus (ToonYou vβ6)
Microbe (DreamShaper v7)
Cthulhu (Epic 2.5D)
Microbe (DreamShaper v7)
Dahlia (Lyriel v1.6)
Microbe (DreamShaper v7)
White Monster (OpenJourney)
Microbe (DreamShaper v7)
Nuclear Explosion (Tenki Hybrid)
Microbe (DreamShaper v7)
White Monster (Counterfeit v3)
Microbe (DreamShaper v7)
White Monster (Counterfeit v3)
Forest (UnstableInkDream v8)
As you can see, the LoRA helps Stable Diffusion get much better at making Goodsell-like images than the base model (as exemplified by the images in Psychedelic Micrographs I). And it's capable of depicting subjects that the real Goodsell is likely not interested in painting. The LoRA isn't perfect, though. Goodsell's images are still much better, so even if he were protective of his style, his job as a molecular biology illustrator would not be in jeopardy.
These illustrations were drawn using Stable Diffusion 1.5 as implemented in SD.Next, with the models indicated above.
The LoRA was trained using kohya_ss in Google colab (https://github.com/bmaltais/kohya_ss). It can be downloaded at https://civitai.com/models/188583/goodsell-style.
